Multimedia data recording apparatus, monitor system, and multimedia data recording method

ABSTRACT

A multimedia data recording apparatus is disclosed that enables large-volume recorded data to be recorded for a long period in accordance with the importance of the data, by means of a simple configuration and simple processing, and without imposing a heavy load on the system. In this apparatus, a layer classification section classifies video data captured by a surveillance camera and input via a video processing section into a plurality of layers on a frame-by-frame basis according to importance. A data recording section assigns a file to each layer and records classified data in a data recording medium. If an information amount reduction section detects that the vacant capacity of the data recording medium has reached or fallen below a threshold value, it deletes a frame belonging to the lowest layer among the plurality of layers.

TECHNICAL FIELD

The present invention relates to a multimedia data recording apparatusand multimedia data recording method used in a monitoring system or thelike.

BACKGROUND ART

In locations requiring all-night or all-day surveillance, such as thestrong rooms of financial institutions or places used to store importantconfidential industrial information, or to give a more familiar example,ATMs (automated teller machines) installed in banks and various kinds ofretail outlets including convenience stores, a surveillance camera,sound-collecting microphone, and so forth, are set up near whatever isbeing monitored, and multimedia data such as continuously captured videodata and voice data can be checked ex post facto when necessary. Withthis kind of surveillance system, the above-described operation isimplemented by storing captured video data, voice data, and the like, ona storage medium (recording medium) such as a hard disk at regularintervals, and reading that data when necessary.

Recently, due to the deteriorating public safety situation in Japan, atrend has appeared of increasing the locations monitored by surveillancesystems, with surveillance cameras being installed in such places asrailway stations, shopping malls, and unmanned warehouses with no inwardor outward movement of goods. A common characteristic of thesesurveillance systems is that, since the objects monitored span a widerange, it has become necessary to install a plurality of surveillancecameras and so forth, and surveillance systems have become larger inscale. The need for higher precision of surveillance data has alsoincreased, and improvements have generally come to be made in video dataresolution and voice data quality. There has thus been a marked increasein the quantity of surveillance data recorded as compared with thesituation heretofore.

However, as there are limits to the capacity of recording media, inorder to perform fixed-period storage of video and other multimediadata, it is necessary for data recorded in the past to be erased afterthe elapse of a fixed period or when the vacant recording mediumcapacity becomes insufficient, and for new data to be recorded in thatvacant space, or for surveillance data to continue to be storedendlessly (hereinafter referred to as “endless recording”) bysequentially overwriting old data with new data.

One conventional multimedia data recording apparatus is a videorecording apparatus that makes it possible to maintain the quality ofimportant video, and also to record video information for a long periodon a medium of limited storage capacity, by storing video informationwhose quality the user wants to ensure in uncompressed form, and storingother video information in compressed form (see Patent Document 1, forexample). This video recording apparatus records video information on avideo recording medium, records the occurrence of a burglary, robbery,traffic accident, fire, or similar incident (hereinafter referred togenerically as an “event”) on the video recording medium, and increasesthe available recording medium capacity by reading from the videorecording medium, from among the video information recorded on therecording medium, video information unrelated to an event, andcompressing and rerecording that read video information.

When an event occurs, both recorded data of the event (hereinafterreferred to as “event data”), and also data immediately prior to theoccurrence of the event (hereinafter referred to as “pre-event data”),are important as material for identifying the cause of the event. Onemultimedia data recording apparatus that performs recording of pre-eventdata (hereinafter referred to as “prerecording” or “pre-eventrecording”) is a surveillance image recording apparatus that savespre-event data in an area separate from the normal recording area (seePatent Document 2, for example). By saving (copying) pre-event data,this apparatus prevents important surveillance data from being erasedafter the elapse of a fixed period.

Patent Document 1: Unexamined Japanese Patent Publication No. 2000-13745

Patent Document 2: Unexamined Japanese Patent Publication No. HEI9-46636

DISCLOSURE OF INVENTION

Problems to be Solved by the Invention

However, a problem with the conventional multimedia data recordingapparatus shown in Patent Document 1 is that a heavy load is imposed onthe system because, when vacant capacity becomes insufficient, secondarymanipulation processing (processing in parallel with recordingprocessing) is performed whereby, while video data output from a monitoror the like is constantly stored in memory, data unrelated to an eventis simultaneously compressed and rerecorded. A large system load alsomeans that the circuit scale of this multimedia data recording apparatusitself is large.

Also, a problem with the conventional multimedia data recordingapparatus shown in Patent Document 2 is that, when pre-event data islong-duration or high-definition data, the amount of data saved (copied)increases, the time necessary for processing and the system load alsoincrease, large-capacity memory with high-speed access capabilitybecomes necessary as the recording medium for saving pre-event data, andsystem costs rise.

It is an object of the present invention to provide a multimedia datarecording apparatus and multimedia data recording method that enablelarge-volume recorded data used in a monitoring system or the like to berecorded for a long period in accordance with the importance of thedata, by means of a simple configuration and simple processing, andwithout imposing a heavy load on the system.

Means for Solving the Problem

In accordance with one aspect of a multimedia data recording apparatusof the present invention, a configuration is employed that includes arecording section that continuously records multimedia data in memory,and a data amount reduction section that, when the vacant capacity ofthe memory is at or below a threshold value, takes older data or lessimportant data among the data recorded in the memory as its object, andreduces the amount of this data.

In accordance with another aspect of a multimedia data recordingapparatus of the present invention, a configuration is employed wherein,in the above-described configuration, the recording section records themultimedia data classified into a plurality of hierarchical layersaccording to the data contents or data precision.

In accordance with yet another aspect of a multimedia data recordingapparatus of the present invention, a configuration is employed wherein,in the above-described configuration, when the vacant capacity of thememory is at or below a threshold value, the data amount reductionsection deletes data in order starting with data recorded in a lowerlayer of the hierarchical layers.

That is to say, according to the present invention, when multimedia datais recorded, this data is recorded classified into hierarchical layersbased on its importance, and when there is little remaining vacantrecording medium capacity, the data amount (information amount) isreduced in order from a low layer. Here, important data denotes datarequiring long-term storage, data requiring high-definition imagequality, high-frame-rate data, data related to an alarm by means of anexternal sensor, and so forth.

According to these configurations, large-volume monitoring data, forexample, is recorded classified into hierarchical layers according tothe importance of that data. Then, when little vacant memory (recordingmedium) capacity remains, deletion is performed in order starting withlow-layer data—that is, data of low importance. Thus, large-volume datacan be recorded for a long period according to its importance, by meansof a simple configuration and simple processing, without imposing aheavy load on a monitoring system or the like.

In accordance with a still further aspect of a multimedia data recordingapparatus of the present invention, a configuration is employed wherein,in the above-described configuration, the data amount reduction sectionperforms deletion in order starting with older data among the datarecorded in a lower layer.

According to this configuration, when deletion is performed in orderstarting with lower-layer data, deletion is performed in order startingwith old data within one particular layer. Thus, looking at recordeddata overall without awareness of layers, a greater amount ofinformation is deleted the older the data is, while the amount ofinformation of new data is maintained. That is to say, the degree ofdeletion of the amount of information varies according to the age of thedata. This means that, while simultaneously considering the two criteriaof “data importance” and “data age,” data of low importance is madesubject to deletion first. At this time, the overall amount of recordeddata can be reduced while maintaining a high-precision state for newdata.

In accordance with a still further aspect of a multimedia data recordingapparatus of the present invention, a configuration is employed wherein,in the above-described configuration, the recording section records themultimedia data classified in a hierarchical structure composed of aplurality of segments set according to the time of acquisition of thisdata, and a plurality of layers that belong to each segment and are setaccording to data contents or data precision; and the data amountreduction section selects a segment whose time of acquisition is older,and performs deletion in order starting with data recorded in a lowerlayer within this segment.

According to this configuration, a segment subject to deletion is firstselected based on the age of the time of recording, and then a layer inwhich data of lower importance is recorded is deleted within thatsegment. Thus, looking at recorded data overall without awareness oflayers, a greater amount of information is deleted the older the datais, while the amount of information of new data is maintained. That isto say, the degree of deletion of the amount of information variesaccording to the age of the data. This means that, while simultaneouslyconsidering the two criteria of “data importance” and “data age,” olddata is made subject to deletion first. At this time, the overall amountof recorded data can be reduced while maintaining a high-precision statefor new data.

Advantageous Effect of the Invention

The present invention enables large-volume recorded data in a monitoringsystem or the like to be recorded for a long period in accordance withthe importance of the data, by means of a simple configuration andsimple processing, and without imposing a heavy load on the system.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a drawing for explaining the installation environment of asurveillance system according to Embodiment 1;

FIG. 2 is a drawing showing a concrete example of recorded data recordedby means of a surveillance system according to Embodiment 1;

FIG. 3 is a drawing showing a concrete example of frame data thatundergoes division into layers on a frame-by-frame basis;

FIG. 4 is a block diagram showing the main configuration of a multimediadata recording apparatus according to Embodiment 1;

FIG. 5 is a flowchart showing the main procedure of data recordingprocessing according to Embodiment 1;

FIG. 6 is a flowchart showing the procedure of information amountreduction processing according to Embodiment 1 in greater detail;

FIG. 7 is a drawing for explaining the relationship between a datamanagement list and data files recorded in a data storage mediumaccording to Embodiment 1;

FIG. 8 is a drawing for explaining how the data configuration varies asa result of information amount reduction processing according toEmbodiment 1;

FIG. 9 is a drawing explaining how the data configuration varies on thetime axis as a result of information amount reduction processingaccording to Embodiment 1;

FIG. 10 is a drawing for explaining variations of recorded data when amultimedia data recording apparatus according to Embodiment 1 performsdata amount reduction processing on MPEG data;

FIG. 11 is a drawing for explaining an overview of data recordingprocessing when MPEG4 FGS is used as an importance criterion;

FIG. 12 is a drawing for explaining differences in image quality whenFGS data divided into layers is played back;

FIG. 13 is a drawing showing concrete examples of data recordingprocessing according to Embodiment 2;

FIG. 14 is a graph of the amount of image variation;

FIG. 15 is a drawing showing the data structure of a data managementlist according to Embodiment 3;

FIG. 16 is a flowchart showing the data recording processing procedureaccording to Embodiment 3;

FIG. 17 is a flowchart showing the procedure of information amountreduction processing according to Embodiment 3 in greater detail;

FIG. 18 is a drawing showing an overview of data recording according toEmbodiment 4;

FIG. 19 is a block diagram showing the main configuration of amultimedia data recording apparatus according to Embodiment 4;

FIG. 20 is a flowchart showing the multimedia data recording processingprocedure according to Embodiment 4;

FIG. 21 is a drawing explaining the data structure in accordance with amultimedia data recording method according to Embodiment 4;

FIG. 22 is a drawing showing a time-sequence representation of theprocedure of a multimedia data recording method according to Embodiment5;

FIG. 23 is a drawing showing concrete examples of Embodiment 5;

FIG. 24 is a block diagram showing the main configuration of amultimedia data recording apparatus according to Embodiment 5;

FIG. 25 is a flowchart showing the processing procedure of a movingsubject determination section according to Embodiment 5;

FIG. 26 is a series of drawings for explaining a concrete example of adata hierarchical structure composed of segments and layers;

FIG. 27 is a series of drawings for explaining a concrete example of adata hierarchical structure composed of segments and layers;

FIG. 28 is a series of drawings for explaining a concrete example of adata hierarchical structure composed of segments and layers;

FIG. 29 is a series of drawings for explaining a concrete example of adata hierarchical structure composed of segments and layers;

FIG. 30 is a series of drawings for explaining a concrete example of adata hierarchical structure composed of segments and layers;

FIG. 31 is a series of drawings for explaining a concrete example of adata hierarchical structure composed of segments and layers;

FIG. 32 is a series of drawings for explaining a concrete example of adata hierarchical structure composed of segments and layers;

FIG. 33 is a series of drawings for explaining a concrete example of adata hierarchical structure composed of segments and layers;

FIG. 34 is a drawing showing a concrete example of Embodiment 6;

FIG. 35 is a block diagram showing the main configuration of amultimedia data recording apparatus according to Embodiment 6;

FIG. 36 is a drawing showing a concrete example of the structure of datarecorded in accordance with a multimedia data recording method accordingto Embodiment 6;

FIG. 37 is a drawing showing a concrete example of Embodiment 7;

FIG. 38 is a block diagram showing the main configuration of amultimedia data recording apparatus according to Embodiment 7; and

FIG. 39 is a drawing showing a concrete example of the structure of datarecorded in accordance with a multimedia data recording method accordingto Embodiment 7.

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will now be described in detailwith reference to the accompanying drawings. In Embodiments 1 through 5,cases where a multimedia data recording apparatus according to thepresent invention is applied to a surveillance system are described byway of example. However, multimedia data recording apparatuses accordingto Embodiments 1 through 5 can also be applied to a more generalmonitoring system. Thus, in Embodiments 6 and 7, cases where amultimedia data recording apparatus according to the present inventionis applied to a monitoring system are described by way of example.

Embodiment 1

FIG. 1 is a drawing for explaining the installation environment of asurveillance system according to Embodiment 1 of the present invention.

In a surveillance system according to this embodiment, it is assumedthat the subject monitored is a dam D1, and one wide-angle camera(surveillance camera C1) is installed and monitors a wide area. Capturedvideo data is stored in a recording medium M1, and is output to anddisplayed on a display terminal T1 as necessary. Characteristics of theobject of surveillance by this system are that there is normallyvirtually no movement, and if there is movement, it is slow. Objects ofsurveillance that have similar characteristics include power generationplants, rivers, the open sea, and so forth.

FIG. 2 is a drawing showing a concrete example of recorded data recordedby means of a surveillance system according to this embodiment.

As the object of surveillance of this surveillance system ischaracterized by virtually no movement, or slow movement if there ismovement, as described above, recorded data essentially merely comprisesa large number of identical still images, as shown in the drawing, eventhough the data is moving-image data.

Thus, in this embodiment, all surveillance data is once stored inrecording medium M1. Then, in view of the characteristics of the objectof surveillance, when little vacant capacity remains, frame thinning-out(frame rate reduction) processing is executed on old video, and a highframe rate used in recording is maintained for new video.

Even video that has undergone frame thinning-out in this way is adequatefor determining an overview of the object of surveillance. Also, sincethere are many cases where recorded video fulfills its role adequatelyas surveillance data even if comprising still images rather than movingimages, it is acceptable for a still image to be formed in effectthrough excessive thinning-out of frames.

It is assumed here that a JPEG or similar format is used as the recordedvideo data format. That is to say, it is assumed that the data is codedusing a coding method that enables compression within a frame. Also,voice data is here assumed to be nontarget data, since speech willbecome discontinuous if subjected to frame thinning-out.

FIG. 3 is a drawing showing a concrete example of frame data thatundergoes division into layers on a frame-by-frame basis.

Each frame data unit (in this drawing, frames 1 through 16) undergoesdivision into layers according to its position on the time axis. To bespecific, taking frame 1 as the base frame, subsequent frames separatedby a 15-frame cycle (frames 1, 16, 31, . . . ) are grouped in layer A.Also, frames separated from frame 1 by a 3-frame cycle (frames 4, 7, 10,. . . ) are grouped in layer B, and frame data belonging to neitherlayer A nor layer B is grouped in layer C.

Thus, the frame rate when recorded data playback is performed using allthe frame data contained in layers A through C can be designated α(fps), the frame rate when playback is performed using all the framedata contained in layers A and B can be designated β (fps), and theframe rate when playback is performed using only the frame datacontained in layer A can be designated γ (fps) (where α>β>γ). That is tosay, the frame rate for playing back recorded data increases in theorder: layer C, layer B, layer A.

FIG. 4 is a block diagram showing the main configuration of a multimediadata recording apparatus 100 according to this embodiment that makes theabove-described operation possible.

Multimedia data recording apparatus 100 has a video processing section101, layer classification section 102, management list creation section103, data management list 104, management list update section 105, datarecording section 106, information amount reduction section 108, dataplayback section 109, and data recording medium M1.

Video processing section 101 executes predetermined video processing onvideo data captured by surveillance camera C1, and outputs the resultingdata to layer classification section 102. Layer classification section102 classifies data output from video processing section 101 into aplurality of layers on a frame-by-frame basis according to importance.Data recording section 106 assigns a file to each layer, and records inorder starting at the start address of data recording medium M1. Dataplayback section 109 reads data from data recording medium M1 asnecessary, and plays back this data via terminal T1. Meanwhile,management list creation section 103 arranges the correspondencesbetween layers and frames decided by layer classification section 102 inlist form, and stores these in data management list 104.

If information amount reduction section 108 detects that the vacantcapacity of data recording medium M1 has reached or fallen below athreshold value, it deletes a frame belonging to the lowest layer amongthe plurality of layers. Management list update section 105 is notifiedby information amount reduction section 108 as to which layer's data hasbeen deleted from data recording medium M1, and based on this, updatesthe correspondences between layers and frames recorded in datamanagement list 104.

Next, the main procedure of data recording processing by a multimediadata recording apparatus with the above-described configuration will bedescribed using the flowchart shown in FIG. 5.

Video data captured by surveillance camera C1 is input to layerclassification section 102 via video processing section 101 (ST1010).The input video data is classified into a plurality of layers on aframe-by-frame basis according to importance (ST1020). For example,assuming that data is divided among three layers, A through C, if framesare input in 1, 2, 3, 4, 5, 6, . . . order, frame 1 is classified as alayer A frame, frame 2 as a layer B frame, and so on.

If there is sufficient vacant capacity to record the above frames indata recording medium M1 (ST1030), data recording section 106 recordsthis frame data in data recording medium M1 (ST1050). Management listcreation section 103 lists the correspondences between layers and framesdecided by layer classification section 102, and stores these in datamanagement list 104 (ST1060).

On the other hand, if there is not sufficient vacant capacity to recordthe above frames in data recording medium M1 (ST1030), informationamount reduction section 108, which constantly monitors the vacantcapacity of data recording medium M1, selects one layer that meets apredetermined condition from the plurality of existing layers, andperforms deletion in order starting with the frame with the oldest timeof recording belonging to that layer (ST1040). The processing in ST1040is repeated until vacant capacity of the predetermined value or more issecured.

Next, the procedure of above-mentioned ST1040 information amountreduction processing will be described in greater detail using theflowchart shown in FIG. 6.

Information amount reduction section 108 first sets the layer subject todeletion to the lowest rank (bottom layer) (ST1041). Then informationamount reduction section 108 confirms whether frame data belongs to thislayer subject to deletion (ST1042), and if belonging data exists,selects the frame data whose time of recording is oldest within thislayer (ST1043), and confirms whether this data has passed the minimumstorage time (ST1044). If the data has passed the minimum storage time,information amount reduction section 108 deletes the frame data selectedin ST1043 (ST1045), and together with this, notifies management listupdate section 105 of the deleted frame data. Management list updatesection 105 updates data management list 104 accordingly (ST1046) If theselected frame data is determined not to have passed the minimum storagetime in ST1044, the layer subject to deletion is raised to the layer onerank higher (ST1047).

The minimum storage time here denotes a period, set beforehand for eachlayer, within which frame data deletion is not performed.

An upper limit may be set for the layer rank increase in ST1047. That isto say, provision can be made so that a layer of a certain rank (levelof importance) or higher is not deleted. As a result, data with thelowest frame rate is not deleted.

FIG. 7 is a drawing for explaining the relationship between datamanagement list 104 and data files recorded in data recording medium M1.

Files of a different area (files #1, #2, #3, . . . ) corresponding tothe respective layers are recorded in data recording medium M1, and thevarious frame data are held in the respective files. For example, frames1, 16, 31, . . . classified as layer A frames are recorded in file #1corresponding to layer A.

At least filenames belonging to the respective layers are recorded indata management list 104 (indicated by arrows in the drawing). In theexample shown in the drawing, information indicating the data sizeoccupied by each layer is also recorded.

FIG. 8A through FIG. 8C are drawings for explaining how the dataconfiguration of data recording medium M1 varies as a result of theabove-described information amount reduction processing.

Assume, for example, that settings have been made so that data isdeleted in the order: layer C, layer B, layer A, and frame data isdeleted starting with the frame data with the oldest time of recording.Assume also that a data minimum storage time has been set for eachlayer, and that a frame that has not passed this minimum storage time isnot deleted.

First, before the above-described information amount reductionprocessing is executed, the data of all layers is arranged on the timeaxis as shown in FIG. 8A. At this time, usage of the recording medium M1disk is 100%, and therefore information amount reduction processing isnecessary. Thus, frames 2, 3, 5, 6, . . . belonging to layer C becomesubject to deletion.

Next, when there are no more frames that can be deleted according to theabove criterion in layer C and the vacant capacity of recording mediumM1 has not yet reached or exceeded the threshold value as shown in FIG.8B (for example, assuming a threshold value of 70%, when the vacantcapacity of recording medium M1 is 67%, as shown in the drawing, andthus less than the threshold value), frames in layer B which is one rankhigher are also deleted. When frames 4, 7, 10, and 13 belonging to layerB are deleted, the situation becomes as shown in FIG. 8C (in thisexample, the vacant capacity of recording medium M1 is 94%).

In a case where, for example, there are no more frames that can bedeleted in layer B, and an upper limit has not been set for layer rankincreases, layer A frames may also be deleted. When layer A frames aredeleted, all the data recorded in that time frame will have beendeleted.

FIG. 9 is a drawing explaining how the data configuration of recordingmedium M1 varies on the time axis as a result of the above-describedinformation amount reduction processing.

As shown in this drawing, the period from time t4 back to time t3 iswithin the layer C minimum storage time, and therefore all the data oflayers A through C remains during this period. Thus, the frame rate atthis time is α (in the drawing, 15 fps). Also, the period from time t3back to time t2 is within the layer B minimum storage time, andtherefore the data of layers A and B remains during this period. Thus,the frame rate at this time is β (in the drawing, 5 fps). Furthermore,the period from time t2 back to time t1 is within the layer A minimumstorage time, and therefore only the data of layer A remains during thisperiod. Thus, the frame rate at this time is γ (in the drawing, 1 fps).

Thus, as time passes, the information amount (frame rate) of recordeddata decreases stepwise, enabling the overall recorded data volume to besuppressed. On the other hand, new data is recorded in a recordingmedium in which spare capacity has arisen due to information amountreduction of old data, and can therefore be recorded for a long period.

Thus, according to this embodiment, surveillance data—which islarge-volume frame data—is stored in recording medium M1 classified intohierarchical layers according to its data structure. That is to say, theminimum data necessary to form a frame is positioned as the mostimportant data, and remaining data has its importance decided accordingto its relative frame rate, and is recorded classified into hierarchicallayers. Then, when little vacant capacity remains in recording mediumM1, deletion is performed in order starting with low-layer data—that is,data of low importance. Thus, large-volume data can be recorded for along period according to its importance, by means of a simpleconfiguration and simple processing, without imposing a heavy load on asurveillance system.

Also, with the above-described configuration, when deletion is performedin order starting with lower-layer data, deletion is performed in orderstarting with old data within one particular layer. Thus, looking atrecorded data overall without awareness of layers, as time passes, agreater amount of information is deleted (the frame rate decreases more)the older the data is, while the amount of information of new data ismaintained. That is to say, the degree of deletion of the amount ofinformation varies according to the age of the data. This means that,while simultaneously considering the two criteria of “data importance”and “data age,” data of low importance is made subject to deletionfirst. At this time, the overall amount of recorded data can be reducedwhile maintaining a high-precision state for new data that enablesmovement of the object of surveillance to be grasped in detail, enablingthe most suitable image to be selected for recognizing a detailed partof the object of surveillance (such as the face of a person being shown,for example) from among these recorded data.

With the above-described configuration, also, by making a setting sothat, in processing that deletes data in order starting with a lowerlayer, deletion is not performed for data that has not passed a minimumstorage time set beforehand for each data, processing is made possiblewhereby the minimum data necessary for grasping an overview of theobject of surveillance—that is, only the most important data consideredfrom the standpoint of data structure—is left (a lower limit is set forthe information amount, and this is maintained), even for comparativelyold data. For new data, on the other hand, a high-precision state ismaintained that enables movement of the object of surveillance to begrasped in detail, enabling the most suitable image to be selected forrecognizing a detailed part of the object of surveillance (such as theface of a person being shown, for example) from among these recordeddata.

In processing that deletes data in order starting with a lower layer, asetting may be made so that a layer of a certain rank (level ofimportance) or higher is not made subject to deletion (an upper limit isset for increasing the rank of the layer subject to deletion). By thismeans, processing is made possible whereby the minimum data necessaryfor grasping an overview of the object of surveillance—that is, only themost important data considered from the standpoint of data structure—isleft (a lower limit is set for the information amount, and this ismaintained), even for old data subject to deletion. Here, theaforementioned lower limit is set for the frame rate, resolution, or thelike.

Even if recorded data is classified into hierarchical layers asdescribed above, when data is deleted, the order of layers to be deletedmay be changed flexibly according to circumstances rather than alwayshaving to delete in order starting with a lower layer. That is to say,layers may be treated as having a flat configuration rather than beingtreated as hierarchical. This enables recorded data to be thought of assimply being grouped focusing on their respective characteristics. Byemploying a configuration such that recorded data are stored classifiedinto a plurality of layers while focusing on their respectivecharacteristics in this way, in addition to processing whereby theinformation amount of recorded data is simply reduced progressivelystarting from data with the oldest time of recording as time passes, itis also possible to reduce the data information amount according to anew deletion criterion not tied to a time criterion. Thus, the deletioncriterion can be amended ex post facto, and a new deletion criterion canalso be added. That is to say, flexibility in reducing the datainformation amount can be improved. This point will be explained againin Embodiment 4.

Here, a case in which the frame rate is used as the criterion fordetermining the importance of data has been described by way of example,but when surveillance data is MPEG (Moving Picture Experts Group) typedata, the MPEG picture type may also be used as a criterion.

To be specific, an I picture (data that enables a still image to beformed with this data only) is classified as layer A, a P picture(difference data between consecutive I pictures, forming a moving imageby supplementing I picture data) as layer B, and a B picture (data forimproving the quality of moving images) as layer C. This will beexplained below in concrete terms using the accompanying drawings.

FIG. 10A through FIG. 10C are drawings for explaining variations ofrecorded data when multimedia data recording apparatus 100 performs dataamount reduction processing on above-described MPEG data. Thismultimedia data recording apparatus has the same basic configuration asthe multimedia data recording apparatus shown in FIG. 4, and therefore adescription thereof is omitted here.

FIG. 10A is a drawing showing a case in which the recorded data isimportant. As shown in this drawing, when data amount reductionprocessing (thinning-out) has not been performed, all I picture, Ppicture, and B picture data belonging to layers A through C is present.In this case, all frames are stored as they are, and therefore when thisdata (that is, MPEG moving image data) is played back, movement issmooth and can be grasped in detail.

However, after layer C deletion has been performed (when only B frameshave been thinned out), only I picture and P picture data remains, asshown in FIG. 10B. Thus, when this data is played back, it has aframe-advance appearance, and movement is jerky (the moving imagequality has deteriorated), but can still be grasped to some extent.

After layer B deletion has also been performed (when P frames have beenthinned out), only I picture data remains, as shown in FIG. 10C. Thus,at this stage, this recorded data cannot form a moving image, and is inthe form of a plurality of recorded still images, being equivalent toJPEG image data.

The recorded video data format is also made to conform to coding thatperforms compression between frames such as MPEG2/MPEG4 and so forth.Therefore, new video or important video is stored as moving-image video.This enables the features of MPEG2/MPEG4 to be exploited.

Unimportant video is changed to a still image storage format. As aresult, the video appears to be stored as a JPEG image, as a still image(thumbnail) representing the situation in a particular time frame.

Thus, even if MPEG picture type is used as a criterion when determiningthe importance of data, the same kind of effect can be obtained as whenthe frame rate is used as a criterion.

Furthermore, an enhancement layer utilized in MPEG scalable coding mayalso be used as a criterion when determining the importance of data.

An MPEG scalable coding method is a hierarchical coding method that, inline with the diversification of networks in recent years, enables videoto be transmitted at quality appropriate to a plurality of bands byhaving a hierarchical structure. MPEG scalable coding methods includeMPEG4 FGS (Fine Granularity Scalability), MPEG4 simple scalable profile,Wavelet based coding, and so forth, but in the following descriptionMPEG4 FGS will be considered as an example.

Video data coded by means of FGS is composed of a base layer, which is amoving image stream for which stand-alone decoding is possible, and atleast one or more enhancement layers, which are moving image streams forimproving the base layer coded moving image quality. The base layer islow-band, low-quality video data, and highly flexible implementation ofhigh image quality can be achieved by supplementing this withenhancement layers according to the band. This will be explained belowin concrete terms using the accompanying drawings.

FIG. 11 is a drawing for explaining an overview of data recordingprocessing when MPEG4 FGS is used as an importance criterion.

Here, we will assume a situation in which it is sufficient to be able tograsp only movement of the object of surveillance, or a situation inwhich only a detailed image of the object of surveillance is necessary.An example would be surveillance of entry into a security zone. In acase of this kind, a low frame rate (low definition) is acceptable forrecorded data when there is nobody in the monitored area. On the otherhand, if an intruder attempts entry, it is only necessary to haveseveral high-definition images (especially images of the intruder'sface).

Here, therefore, as shown in FIG. 11, the FGS base layer is mapped ontolayer A and the FGS enhancement layers are mapped onto layer B and layerC respectively. By this means, it is possible to change not only theframe rate but also the resolution (image quality) of important data.For other data, on the other hand, the frame rate is kept high enough toenable human movements to be grasped even if the resolution is low.Also, if an abnormality occurs, such as attempted entry that does notfollow the prescribed procedure, for example, data is saved just as itwas when recorded (written) (that is, at a high frame rate and highresolution).

FIG. 12A through FIG. 12D are drawings for explaining differences inimage quality when FGS data divided into layers as described above isplayed back. FIG. 12D shows data at the time of recording—that is, theimage quality when data with layers A through C all fully included isplayed back; FIG. 12A shows the image quality when the FGS enhancementlayers (layers B and C in terms of the layer divisions of the presentinvention) have been deleted and only the FGS base layer (layer A)remains; FIG. 12B shows the image quality when the FGS “motionimproving” enhancement layer (layer C) has been deleted and layer A andlayer B remain; and FIG. 12C shows the image quality when the FGS “imagequality improving” enhancement layer (layer B) has been deleted andlayer A and layer C remain.

In FIG. 12B, since the FGS “motion improving” enhancement layer has beendeleted, the played-back image is a moving image with motion, but withpoor precision and a frame-advance appearance. On the other hand, inFIG. 12C, since the FGS “image quality improving” enhancement layer hasbeen deleted, motion is smooth but image quality is poor, and theoutline of the person's face is jagged. In FIG. 12A, since both the FGS“motion improving” enhancement layer and the FGS “image qualityimproving” enhancement layer have been deleted, the video has aframe-advance appearance and poor image quality.

Thus, even if enhancement layers used in MPEG scalable coding are usedas a criterion when determining the importance of data, the same kind ofeffect can be obtained as when the frame rate is used as a criterion.

Embodiment 2

One method of setting a criterion for determining the importance of datais to focus on the data structure and consider data important to theextent that it is minimal necessary data for showing certain information(such as I picture data in the case of MPEG, for example), asillustrated in Embodiment 1. However, the importance of the contentitself of information indicated by data can also be used as a dataimportance criterion. For example, data recorded before and after theoccurrence of an event, or data with a large amount of image variation(=vigorous movement) between frames, can be said to be important data.

In this embodiment, an object of surveillance is supposed for whichthere is normally no movement, or for which, if there is movement,scenes in which there is movement and there is no movement can beclearly distinguished. For example, in the case of surveillance of anoutdoor installation or surveillance of a building at night, monitoringaccess to a specific room or the like conforms to this scenario.

A characteristic of these surveillance data is that there are a numberof successive image frames that can virtually be classed together withinrecorded data. These data cannot be called important data since howevermany are stored the storage area cannot be said to be used effectively.Therefore, in this embodiment, layers are decided according to theamount of image variation (the degree of vigor of movement of the objectof surveillance). For example, if the difference in brightness comparedwith the previous frame is great (image variation is great) data isclassified as layer A data, and if the difference in brightness comparedwith the previous frame is small (image variation is small) data isclassified as layer C data. Then reduction of the data amount by meansof frame thinning-out or the like is performed on layer C, etc. On theother hand, for old recorded data, frames with a small amount of imagevariation between frames are thinned out. This will be explained belowin concrete terms using the accompanying drawings.

FIG. 13A and FIG. 13B are drawings showing concrete examples of theabove-described data recording processing.

When a surveillance camera is not showing anything, as shown in FIG.13B, there is little value in storing recorded data for a long period.However, when the surveillance camera shows an intruder I, it is highlynecessary to store this recorded data. To consider the amounts of imagevariation of these data using FIG. 14, in the image variation amountgraph in FIG. 14 the kind of recorded data shown in FIG. 13A is data P2showing the highest peak. Thus, as shown in FIG. 14, this data isclassified as layer A data, data P1 with a lower peak than data P2 isclassified as layer B data, and remaining data not showing a peak isclassified as layer C data.

Thus, according to this embodiment, surveillance data—which islarge-volume frame data—is stored in recording medium M1 classified intohierarchical layers according to the amount of image variation betweenframes. That is to say, recording is performed in accordance with theimportance of the content indicated by the data. Then, when littlevacant capacity remains in recording medium M1, deletion is performedstarting with data of low importance. Thus, large-volume data can berecorded for a long period according to its importance, by means of asimple configuration and simple processing, without imposing a heavyload on a surveillance system.

When data recorded after the occurrence of an event is considered to beimportant data, it is possible, for example, for data recorded after theoccurrence of an event of high importance in terms of event content tobe classified as layer A data, for data of an event of low importance asan event in comparison with data classified as layer A data to beclassified as layer B data, and for data not related to an event butrecorded periodically in accordance with scheduling to be classified aslayer C data.

Embodiment 3

In this embodiment, the data structure of data management list 104 ismade a further subdivided hierarchical structure. That is to say, a“segment” layer is further provided above the layers described inEmbodiment 1 and Embodiment 2, and recorded data is recorded within thisfurther subdivided structure.

FIG. 15 is a drawing showing the data structure of data management list104.

In this embodiment, a group of frames recorded in a fixed period istaken as data of one segment, and a level indicating data importance isset on a segment-by-segment basis. The (rank of the) layer stored isdecided according to this level. That is to say, when recording, layersare assigned according to importance in segment units, and in dataamount reduction, layers of a certain fixed rank or higher are kept, andlayers of ranks lower than this level are deleted.

As the criterion for determining the importance of data, the importanceof the content itself of the information indicated by the data is used,in the same way as in Embodiment 2. To be specific, a level is decidedaccording to the importance of the factor (event, schedule, userinstruction, etc.) responsible for recording the data. For example, acase where a sensor detects an abnormality and issues an alarm isconsidered to be an important event. That is to say, in a third-partyintrusion prevention system, data collected “when a human-body sensorreacts” or “when a person perceives an abnormality and sounds an alarmbell” is important, and is therefore treated as of the highestimportance. On the other hand, “no level” is set for data unrelated toan event (such as data recorded periodically in accordance withscheduling, for example).

When information amount reduction section 108 deletes a “no level”segment, it deletes all the layers within that segment. On the otherhand, if a layer of a level lower than the level of the segment subjectto deletion is held in this segment, only frames belonging to this layerare deleted. If the segment level and stored layer level are the same,the segment subject to deletion is switched to the next segment.

FIG. 16 is a flowchart showing the data recording processing procedureaccording to this embodiment. This flowchart is identical to theflowchart shown in FIG. 5 up to ST1050, and therefore a description ofthis part of the flowchart is omitted here.

After frame data is recorded in ST1050, it is determined whether theoccurrence of an event has been reported by means of an alarm from anexternal sensor or the like (whether there is event input) (ST3010).Then, if an event has occurred, the layer in which the frame data is tobe protected is decided according to the type of event (ST3020).

For example, if the water level of dam D1 has merely exceeded theaverage level, a comparatively low layer is assigned to the data inwhich this fact is recorded, but if a maximum-level accident hasoccurred, such as when the water level of dam D1 has exceeded a warninglevel or the wall of dam D1 has burst, the highest layer is assigned tothat frame data.

After the layer has been decided according to the type of event inST3020, or if it is determined in ST3010 that there is no external eventinput, the management list updating described earlier is performed(ST1060).

FIG. 17 is a flowchart showing the procedure of information amountreduction processing (ST1040) according to this embodiment in greaterdetail.

Information amount reduction section 108 first sets the data subject todeletion to the segment with the oldest time of recording (ST3110). Theninformation amount reduction section 108 confirms whether a layer forwhich deletion is possible is present in this segment (ST3120), and ifan applicable layer is present, selects the frame data with the oldestrecording time within this layer (ST3130), and deletes this frame data(ST3140). Information amount reduction section 108 also notifiesmanagement list update section 105 of this deleted frame data.Management list update section 105 updates data management list 104 inaccordance with this notification (ST3150). If it is determined inST3120 that there is no layer for which deletion is possible, thesegment subject to deletion is set to the next oldest segment (ST3160),and the processing flow returns to ST3120.

A case has been described here, as an example, in which a segment is setaccording to the age of the time of recording, but this embodiment isnot limited to this case, and data may also be divided into segmentsbased on the importance of the data (where this is a different criterionfrom the criterion for layer division).

Thus, according to this embodiment, recorded data is recorded in afurther subdivided hierarchical structure composed of segments andlayers, enabling data to be erased based on a plurality of deletioncriteria, and a more flexible data recording method to be provided.Also, by this means, the degree of information amount (data amount)deletion can be varied according to the importance of data. Generally,even in the case of so-called important data, the precision of dataconsidered as important comes to differ in the future according to theproperties of the data. For example, even if data is classified asimportant data because it is related to an event, the required dataprecision varies according to the degree to which a user wantsinformation related to the content of that event. That is to say, thefact that data is important data does not necessarily mean that all thedata is considered necessary. On the other hand, even if data isclassified as important data, when the properties of surveillance dataare considered, rather than deleting all recorded data because it hasbecome old, it is desirable to store part of the data at fixed intervalssince it does not matter if the data is of low resolution or lowprecision. Therefore, in this embodiment, a versatile data recordingmethod is provided in consideration of the above.

Also, with the above-described configuration, a segment subject todeletion is first selected based on the age of the time of recording,and then a layer in which data of low importance conforming to thedeletion conditions is stored within that segment is deleted. Thus,looking at recorded data overall without awareness of layers, as timepasses, a greater amount of information is deleted the older the datais, while the amount of information of new data is maintained. That isto say, the degree of deletion of the amount of information variesaccording to the age of the data. This means that, while simultaneouslyconsidering the two criteria of “data importance” and “data age,” olddata is made subject to deletion first. At this time, the overall amountof recorded data can be reduced while maintaining a high-precision statefor new data that enables movement of the object of surveillance to begrasped in detail, enabling the most suitable image to be selected forrecognizing a detailed part of the object of surveillance (such as theface of a person being shown, for example) from among these recordeddata.

Embodiment 4

In this embodiment, the layer of data recorded before and after theoccurrence of an event (event data and pre-event data) can be changed expost facto. This is because, in a surveillance system installed at anintersection, for example, a recording prior to the occurrence of atraffic accident is more important than a recording after the occurrenceof the accident for the purpose of analyzing the cause of the accident.This will be explained below in concrete terms using the accompanyingdrawings.

FIG. 18 is a drawing showing an overview of data recording according tothis embodiment.

In this surveillance system, at time t4 during surveillance a sensordetects the occurrence of an abnormal state (event) subject to an alarm,and issues an alarm to the user, etc., based on this. Meanwhile, amultimedia data recording apparatus according to this embodimentperforms video recording linked to this alarm issuance.

Video captured from the occurrence of the event onward (event data) isimportant data, but video (pre-event data) V1 captured at time t1through time t3 prior to the occurrence of the event is also importantdata. Therefore, this pre-event data (long-duration data spanningseveral seconds to several tens of minutes) is also recorded.

When an event occurs, the rank of the layer of data to be recordedhenceforth is raised above that for normal recording. Also, sincepre-event data is also important and requires the same high-definitionimage quality and high frame rate as event data, when an event occurs, achange is also implemented that raises the layer rank (importance level)of already recorded data ex post facto. That is to say, going back to apredetermined time in the past from the time of the event occurrence, achange is implemented that raises the rank of the layer in which datarecorded from that point in time is classified. At this time, only thelayer rank is changed, and data movement (a change of the data recordinglocation in memory) is not performed.

The layer rank may also be decided taking the type of event (theimportance of the event itself) into consideration, in the same way asshown in Embodiment 3.

FIG. 19 is a block diagram showing the main configuration of amultimedia data recording apparatus 400 according to this embodimentthat makes the above-described operation possible. This multimedia datarecording apparatus 400 has a similar basic configuration to multimediadata recording apparatus 100 shown in Embodiment 1 (see FIG. 4), andtherefore identical configuration elements are assigned the same codesas in FIG. 4 and descriptions thereof are omitted.

Multimedia data recording apparatus 400 has an event input section 401and a management list change section 402. On receiving an alarmnotification from an external sensor S1, event input section 401 reportsthis to management list change section 402, and changes the contents ofdata management list 104.

The procedure of the above-described multimedia data recordingprocessing will now be described using the flowchart shown in FIG. 20.The procedure of this flowchart is basically similar to that of theflowchart shown in FIG. 16, and therefore identical steps are assignedthe same codes and a description of these steps is omitted here.

After a layer has been decided according to the type of event in ST3020,it is determined whether pre-event recording is necessary (ST4010).Specifically, it is determined whether or not an alarm has been reportedto event input section 401 from external sensor S1. Then, if pre-eventrecording is necessary, going back to a predetermined time in the pastfrom the time of the event occurrence, a change is implemented thatraises the rank of the layer in which data recorded from that point intime is protected (classified) (ST4020). The final saved datainformation amount is decided according to this layer rank. Thenmanagement list change section 402 updates the contents recorded in datamanagement list 104 in accordance with the above-described layer change(ST4030) If it is determined in ST3010 that there is no external eventinput, above-described processing steps ST3020, ST4010, and ST4020 arenot performed.

FIG. 21 is a drawing explaining the data structure in accordance with amultimedia data recording method according to this embodiment.

A multimedia data recording apparatus according to this embodimentdecides the layer to be protected and the data update range at the pointin time at which the respective event is input. In the example in thisdrawing, event A and event B are input, data D401 is data belonging to alayer to be protected for event A (event data and pre-event data), dataD403 is data belonging to a layer to be protected for event B (eventdata and pre-event data), and data D402 is determined to be data forwhich deletion is possible in the future. Therefore, data D402 isdeleted immediately when the vacant capacity of memory M1 falls to orbelow a predetermined value (when capacity is exceeded) or after theelapse of a predetermined time. This data D402 is stored all together(in consecutive memory addresses) to facilitate deletion.

FIG. 22 is a drawing showing a time-sequence representation of theprocedure of a multimedia data recording method according to thisembodiment.

As can be seen from this drawing, the multimedia data recording methodaccording to this embodiment is composed of three stages. That is tosay, when multimedia data is acquired, this data is stored divided intolayers, and an index for managing the data hierarchically and datacharacteristics—that is, information on recording quality and so forth(actually, information specifying the protection level of each layersince division into layers is performed on a quality-by-qualitybasis)—are also registered. Then, when an event occurs, the event dataand pre-event data recording quality—more specifically, the layerprotection level—is changed. Then, when the vacant capacity of memory M1falls to or below a predetermined value, or after the elapse of apredetermined time, deletion is started from a low layer. Thus, databelonging to layers other than layers subject to protection is deleted.

Thus, according to this embodiment, when an event occurs, going back intime the rank of the layer in which pre-event data is stored is changedex post facto, enabling data stored in this layer to be recorded for along period and at high definition.

Moreover, with the above-described configuration, only the rank of alayer is changed in processing for long-duration recording of pre-eventdata, so that there is no need for a separate recording medium forsaving pre-event data temporarily. The circuit scale of the datarecording apparatus can thus be reduced.

Furthermore, with the above-described configuration, classification intoa hierarchical layer is performed when surveillance data is recorded,and the rank of this layer can be changed ex post facto, enablingflexibility of data recording processing to be improved.

Embodiment 5

In this embodiment, a moving subject is tracked within video shot by acamera, and when the nature of movement of that moving subject matches apredetermined condition this is taken to be an event, and data recordedfrom back when movement started up to the present can be stored at highdefinition.

To consider the concrete example of this embodiment shown in FIG. 23,for instance, in a surveillance system installed near the entrance to anapartment building, detecting a person showing suspicious behavior whenentering an apartment and storing the video for that time at highdefinition enables the detailed behavior of the suspicious person to beconfirmed ex post facto. Specifically, when a person being trackedenters a specified area, video data from immediately before entry untilthe person disappears is stored at low quality, and when a person beingtracked remains in a specified area for a long period, video data fromthe person's appearance in the video until the person' disappearancefrom the video is stored at high quality.

FIG. 24 is a block diagram showing the main configuration of amultimedia data recording apparatus 500 according to this embodiment.This multimedia data recording apparatus 500 has a similar basicconfiguration to multimedia data recording apparatus 400 shown inEmbodiment 4 (see FIG. 19), and therefore identical configurationelements are assigned the same codes as in FIG. 19 and descriptionsthereof are omitted. Also, in the case of configuration elements whosebasic operation is the same but that differ slightly in detail, the samereference numbers are assigned with a lower-case letter appended, andadditional description is provided as appropriate.

Multimedia data recording apparatus 500 has a moving subjectdetermination section 501, and creates or changes data management list104 according to the nature of the behavior of a moving subject. Movingsubject determination section 501 tracks a moving subject within videoshot by a camera, determines whether or not that moving subject iswithin a predetermined area, and furthermore, if that moving subject iswithin the predetermined area continuously, calculates the duration ofthat continuous presence.

Details of the determination made by moving subject determinationsection 501, and operations based on that determination, will now bedescribed.

If moving subject determination section 501 determines that a movingsubject is not present in the video, or is present but is not within apredetermined area, a management list creation section 103 a creates adata management list 104 with the level indicating the importance of thesegment to which frame data recorded by data recording section 106belongs as “no level.” By this means, data of all layers is deletedduring data amount reduction, and the segment is completely deleted.

On the other hand, if it is determined by moving subject determinationsection 501 that there is a moving subject within the predeterminedarea, management list creation section 103 a creates a data managementlist 104 with the level indicating segment importance raised. By thismeans, some layer data is deleted according to the level at the time ofdata amount reduction, and the overall segment data amount is reduced.

When it is determined by moving subject determination section 501 that amoving subject is within the predetermined area, and furthermore thetime during which the moving subject has remained continuously exceeds apredetermined time, management list creation section 103 a sets thelevel indicating the importance of segments to be recorded thereafter tothe highest level, and creates a data management list 104. Also,management list change section 402 a changes data management list 104 sothat the level indicating the importance of segments recorded from backwhen tracking of the moving subject started up to the present is madethe highest level. By this means, all layers are saved, and not erased,for all segments from the start of tracking of the moving subject untiltracking ends.

The detailed processing procedure of moving subject determinationsection 501 will now be described using the flowchart shown in FIG. 25.

Moving subject determination section 501 first determines whether or nota moving subject is present in the video shot by surveillance camera C1(ST5010). For example, memory is provided in which immediately precedingframe data is stored constantly, a difference comparison is performedbetween the immediately preceding frame in memory and the current frame,and if the difference in brightness values exceeds a predeterminedthreshold, it is determined that a moving subject is present.Furthermore, the certainty of the moving subject being a person may beincreased based on the size and shape of the difference area, andcontinuity of movement in a comparison of preceding and succeedingframes. Next, if a moving subject is present in the video, it isconfirmed whether or not the current mode is moving subject trackingrecording mode (ST5020). Moving subject tracking recording mode denotesa state in which a moving subject is continuously present in apredetermined area for a predetermined time. If moving subject trackingrecording mode is confirmed, notification is given to management listcreation section 103 a to create a data management list 104 with thelevel indicating the importance of the current segment made the highestlevel (ST5030).

On the other hand, if the mode is determined not to be moving subjecttracking recording mode in ST5020, moving subject coordinates arecalculated (ST5040). Moving subject coordinates are the coordinates ofthe barycenter of the aforementioned difference area. It is thendetermined whether or not the moving subject is within the predeterminedarea (ST5050). The predetermined area is an area in which the object ofsurveillance such as a bank entrance or an ATM is present, and isselected by the surveillance operative beforehand on a monitor screen.If the moving subject is within the predetermined area, the time duringwhich the moving subject is present continuously within thepredetermined area is calculated (ST5060), and it is determined whetheror not the duration of that continuous presence exceeds a predeterminedtime (ST5070). If the duration of continuous presence exceeds thepredetermined time, notification is given to management list creationsection 103 a to create a data management list 104 with the levelindicating the current importance made the highest level (ST5080).

Furthermore, notification is given to management list change section 402a to change the segment level from the time when a moving subject wasfirst determined to be present in the video (the moving subject trackingstart time) up to the present to the highest level (ST5090). Then movingsubject tracking recording mode is initiated in order to make the levelof data management list 104 segments to be created subsequently thehighest level (ST5100). If it is determined in ST5070 that the durationof continuous presence has not exceeded the predetermined time,notification is given to management list creation section 103 a to raisethe current segment level (ST5140).

On the other hand, if it is determined in ST5050 that a moving subjectis not present within the predetermined area, or if it is determined inST5010 that a moving subject is not present in the video, notificationis given to management list creation section 103 a to set the currentsegment level to “no level” (ST5120), and moving subject trackingrecording mode is cleared (ST5130).

The processing procedure for data amount reduction is similar to that inEmbodiment 3, and therefore a description thereof will be omitted here.

FIG. 26 through FIG. 33 are series of drawings for explaining concreteexamples of a data hierarchical structure composed of segments andlayers. FIG. 28 through FIG. 33 are drawings showing the actualconfiguration of data that manages individual segments.

For example, if reference numbers are assigned as shown in FIG. 26 tosurveillance data of scenes of the same kind as shown in FIG. 23,information indicating the characteristics of each data is as shown inFIG. 27, according to whether or not there is a moving subject andwhether or not there is event occurrence. As shown in this drawing, thepresence of an intruder is detected in D524, and the fact that theduration of continuous presence of the intruder has exceeded thepredetermined time is detected in D525.

As shown in FIG. 28, in data D521 through D523 “none” is entered as theprotected layer entry as there is no event. Also, as shown in FIG. 29,in data D524 the protected layer is changed to A from the segment 5seconds before since there is an intrusion event. Furthermore, as shownin FIG. 30, in data D525 the protected layers of “moving subjectpresent” segments are changed to “A, B”.

On the other hand, as shown in FIG. 31, in D526 data belonging to thelowest layer that is not a protected layer (layer C) is deleted. Also,as shown in FIG. 32, in D527 all layers C are deleted, and thereforelayer B of the ID=1 segment is deleted next. Lastly, in D528 layer A ofthe ID=1 segment is deleted, and all layers of the ID=1 segment aredeleted, and therefore the management data of the ID=1 segment isdeleted.

Thus, according to this embodiment, it is possible for video in whichthere is no moving subject to be stored only temporarily, for video atthe time when a moving subject enters a predetermined area to be storedas low-resolution video, and furthermore, when the moving subjectremains in the predetermined area for a predetermined time or longer,for the entire video from the appearance to the disappearance of themoving subject to be stored as high-resolution video.

By this means, when a predetermined area setting is made for theentrance of a high-security room where personal verification (input of apersonal identification number or the like) is implemented upon entry,for example, video of a person entering smoothly can be stored at aresolution sufficient to be able to identify the person, and video of aperson who takes time to enter can be stored at high resolution thatenables the face and movements of the person to be checked in detail.

Embodiment 6

In this embodiment, video played back frequently can be stored at highdefinition, video not played back so much can be stored with the amountof data reduced, and video not played back at all can be completelydeleted.

For example, to consider the concrete example of this embodiment shownin FIG. 34, in a monitoring system in which not only real-time videoshot by a camera installed in a sightseeing area or the like, but alsorecorded past video, can be displayed on a PC or mobile terminal via theInternet or the like, video for which there are many requests forplayback by the user—in FIG. 34, for example, video shot under brightconditions of daylight and fine weather (D601) or video shot when anevent of some kind is in progress (D603)—is highly likely to be playedback subsequently, and is therefore stored at high definition. On theother hand, video that is not played back by the user at all—in FIG. 34,for example, video shot at night (D602) or video shot in the daytime butwhen it is raining (D604)—will represent simply a waste of recordingcapacity if stored, and therefore is subjected to either reduction ofthe amount of data or complete deletion, thereby securing capacity forrecording newly shot video.

FIG. 35 is a block diagram showing the main configuration of amultimedia data recording apparatus 600 according to this embodiment.This multimedia data recording apparatus 600 has a similar basicconfiguration to multimedia data recording apparatus 400 shown inEmbodiment 4 (see FIG. 19), and therefore identical configurationelements are assigned the same codes as in FIG. 19 and descriptionsthereof are omitted. Also, in the case of configuration elements whosebasic operation is the same but that differ slightly in detail, the samereference numbers are assigned with a lower-case letter appended, andadditional description is provided as appropriate.

In multimedia data recording apparatus 600, a data playback section 109b reads data from recording medium M1, plays back this data via displayterminal T1, and also reports this data to a management list changesection 402 b, and management list change section 402 b finds thesegment corresponding to the played-back data from data management list104, and raises the level indicating the importance of that segment.

The processing procedure of multimedia data recording apparatus 600 willnow be described.

A management list creation section 103 b creates a data management list104 with the level indicating the importance of the segment to whichframe data recorded by data recording section 106 belongs as “no level.”By this means, the relevant segment undergoes deletion of data of alllayers during data amount reduction, and is completely deleted.

Data playback section 109 b reads the relevant data from recordingmedium M1 in accordance with a video playback request from the user, andalso sends identification information for the read data (frame number ortime information) to management list change section 402 b. Based on thisidentification information, management list change section 402 b findsthe segment to which the read data belongs from data management list104. In data management list 104 the number of playback times is storedon a segment-by-segment basis, and the number of playback times of thefound segment is totaled. If the number of playback times exceeds apredetermined numeric value at this time, the level indicating theimportance of the aforementioned segment is raised by 1.

The processing procedure for data amount reduction is similar to that inEmbodiment 3, and therefore a description thereof will be omitted here.

FIG. 36 is a drawing showing a concrete example of the structure of datarecorded in accordance with a multimedia data recording method accordingto this embodiment.

D611 is management data when video is stored. In D612, there is aviewing request for stored video, and therefore the number of timesviewed is totaled. In D613, the number of times viewed exceeds apredetermined threshold, and therefore A is added to the layers subjectto protection. In D614, the number of times viewed further increases,and the layers subject to protection increase. Details are the same asfor the data structure described in other embodiments, and are thereforeomitted here.

Thus, according to this embodiment, the greater the number of playbackrequests from a user the higher is the definition used for storingrecorded video, and video for which there are no playback requests atall from a user can be deleted or stored at low definition, enabling arecording method to be provided that responds to user needs.

It is also possible for the final playback time to be stored in datamanagement list 104, and for management list change section 402 to bechanged so that, even for a segment with a large number of playbacktimes, the level indicating the importance of the relevant segment islowered by 1 or changed to “no level” if a predetermined time or longerhas elapsed since the last playback.

By this means, video for which there have been many playback requestsfrom the user but for which there has been no playback request for along period can be regarded as unnecessary video, and deleted or storedwith its definition lowered.

Embodiment 7

In this embodiment, from among a plurality of videos shot and recordedat different times under the same conditions of camera location anddirection, one video matching a predetermined condition can be stored athigh definition while the other videos are stored with the amount ofdata reduced.

For example, to consider the concrete example of this embodiment shownin FIG. 37, with regard to videos shot with a digital still camera,video camera, or the like, in sightseeing areas, a plurality of recordedvideos (D702, D705, D707) shot and recorded under the same shootingconditions (camera location, direction, and so forth) are designated as“video with duplicated content”, and of these videos, most recent videoD707 is always stored as high-quality video, while D702 and D705 arestored with the amount of data reduced in proportion to how far backthey are in the past. Reducing the amount of data of video withduplicated content prior to storage in this way enables a large amountof video of various content to be stored.

FIG. 38 is a block diagram showing the main configuration of amultimedia data recording apparatus 700 according to this embodiment.This multimedia data recording apparatus 700 has a similar basicconfiguration to multimedia data recording apparatus 400 shown inEmbodiment 4 (see FIG. 19), and therefore identical configurationelements are assigned the same codes as in FIG. 19 and descriptionsthereof are omitted. Also, in the case of configuration elements whosebasic operation is the same but that differ slightly in detail, the samereference numbers are assigned with a lower-case letter appended, andadditional description is provided as appropriate.

Multimedia data recording apparatus 700 has a camera informationacquisition section 701, acquires camera information comprising thecamera shooting location obtained by means of GPS or the like and thecamera shooting direction obtained by means of a magnetic field sensoror the like, stores this together with video shot by the camera, and ifvideo that has the same camera information has been stored in the past,reduces the amount of data of video with an older time of recordingsince duplicate storage has been performed of video with the samecontent.

The processing procedure of multimedia data recording apparatus 700 willnow be described.

In this embodiment, a group of frames shot and recorded consecutivelywith the same camera attributes are treated as data of one segment.Camera attribute information is entered in data management list 104 on asegment-by-segment basis.

Camera information acquisition section 701 periodically acquires camerainformation comprising the camera shooting location obtained by means ofGPS or the like and the camera shooting direction obtained by means of amagnetic field sensor or the like, and if either the shooting locationor the shooting direction in the acquired shooting information isdifferent from that acquired the previous time, notifies a managementlist creation section 103 c that there has been a change in the camerainformation, and also sends camera attribute information to a managementlist change section 402 c.

On receiving notification that the camera information has changed fromcamera information acquisition section 701, management list creationsection 103 c creates a data management list 104 so that subsequentlyrecorded frame data belongs to a different new segment, and also setsthe level indicating the importance of that segment to the highestlevel. By this means, unless there is a subsequent change in level, allsubsequent layers are saved, and not deleted, for that segment.

On receiving camera information from camera information acquisitionsection 701, management list change section 402 c finds a segment forwhich the same shooting location and shooting direction are stored as inthat camera information from data management list 104, and also lowersthe level indicating the importance of the found segment by 1, or sets“no level” if the lowest level is already set.

FIG. 39 is a drawing showing a concrete example of the structure of datarecorded in accordance with a multimedia data recording method accordingto this embodiment.

In D711, all layers (A through C) are made subject to protection whenvideo storage is performed. Then, in D712, when video (a segment) thathas the same camera information is stored, layer C is removed from thelayers subject to protection. Furthermore, in D713, when video that hasthe same camera information is stored numerous times, the indication oflayers subject to protection is changed to “none.” Details are the sameas for the data structure previously described in other embodiments, andare therefore omitted here.

Thus, according to this embodiment, from among a plurality of videosshot and recorded using the same camera location and direction, the mostrecent video is stored at high definition, other video is stored withdata amount reduction performed in proportion to how far back in thepast that video is, and video further in the past is deleted. By thismeans, a user can be spared the time and trouble of manually finding anddeleting video with the same content, and a large amount of video shotat various locations can be stored.

This concludes the descriptions of the embodiments of the presentinvention.

A multimedia data recording apparatus and multimedia data recordingmethod according to the present invention are not limited to theabove-described embodiments, and various variations and modificationsmay be possible without departing from the scope of the presentinvention. For example, it is possible to implement the presentinvention by combining embodiments as appropriate.

Also, in the above description, a case has been described by way ofexample in which a multimedia data recording apparatus according to thepresent invention is applied to a surveillance system—that is, theapparatus records surveillance data—but the object of recording is notlimited to this.

Furthermore, a case has here been described by way of example in whichthe present invention is configured by means of hardware, but it is alsopossible for the present invention to be implemented by means ofsoftware.

The present application is based on Japanese Patent Application No.2003-360012 filed on Oct. 20, 2003, and Japanese Patent Application No.2004-306093 filed on Oct. 20, 2004, the entire content of which isexpressly incorporated by reference herein.

INDUSTRIAL APPLICABILITY

A multimedia data recording apparatus according to the present inventionenables data to be recorded for a long period by means of a simpleconfiguration and simple processing without imposing a heavy load on thesystem, and is thus suitable for use in a surveillance system or thelike.

1. A multimedia data recording apparatus comprising: a layerclassification section that classifies multimedia data into a pluralityof hierarchical layers according to data contents or data precision; arecording section that continuously records data classified into saidlayers in memory; and a data amount reduction section that, when vacantcapacity of said memory is at or below a threshold value, performsdeletion in order starting with data classified into a lower layer ofsaid hierarchical layers among data recorded in said memory. 2.(canceled)
 3. The multimedia data recording apparatus according to claim1, wherein said layer classification section classifies said multimediadata into a plurality of hierarchical layers according to frame rate,required image quality or resolution, image variation amount betweenframes, required storage time, MPEG (Moving Picture Experts Group) datapicture type, importance of a recorded event, or an enhancement layer ofdata coded by an MPEG scalable coding method.
 4. (canceled)
 5. Themultimedia data recording apparatus according to claim 1, wherein saiddata amount reduction section performs deletion in order starting witholder data or less important data among data recorded in a lower layer.6. The multimedia data recording apparatus according to claim 5, whereinsaid data amount reduction section does not delete data that has notpassed a minimum storage time among data recorded in a lower layer. 7.The multimedia data recording apparatus according to claim 1, wherein:said layer classification section sets said multimedia data in aplurality of segments according to a time of acquisition of this dataand then classifies said multimedia data into said layers for each saidsegment; and said data amount reduction section selects a segment whosesaid time of acquisition is older, and performs deletion in orderstarting with data recorded in a lower layer within this segment.
 8. Amonitoring system equipped with the multimedia data recording apparatusaccording to claim 1, said monitoring system further comprising adetection section that detects event occurrence in a monitored area;wherein said data amount reduction section excludes at least one or aplurality of layers in which data related to said event is recorded fromdata amount reduction as a protected layer.
 9. A monitoring systemcomprising: the multimedia data recording apparatus according to claim1; a measuring section that measures frequency of access to datarecorded in said memory; and a change section that changes at least oneor a plurality of layers in which data whose frequency of access isgreater than or equal to a predetermined value among data recorded insaid memory to a protected layer that is not subject to data amountreduction by said data amount reduction section.
 10. A monitoring systemcomprising: the multimedia data recording apparatus according to claim1; and a determination section that determines mutual similarity of datarecorded in said memory; wherein said data amount reduction sectionperforms deletion in order starting with data recorded in a lower layerfor a layer in which older data is recorded among data determined tohave a high degree of similarity by said determination section.
 11. Amultimedia data recording method comprising: a layer classifying step ofclassifying multimedia data into a plurality of hierarchical layersaccording to data contents or data precision; a recording step ofcontinuously recording data classified into said layers in memory; and adata amount reducing step of, when vacant capacity of said memory is ator below a threshold value, taking older data or less important dataamong data recorded in said memory as its object, and reducing a dataamount of these data.